Papers with Evaluating large language models
From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics for large language models yield numerical scores that ignore user experience. |
| Approach: | They propose a metric that suggests revision edits that mimic the human writing process . their results show that the metric offers more insightful feedback and distinguishes between texts . |
| Outcome: | The proposed metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. |
LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation (2025.findings-emnlp)
Copied to clipboard
Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Current medical benchmarks have limitations in question design, data sources and evaluation methods. |
| Approach: | They propose a new benchmark covering five core medical areas . it includes 2,996 questions created from real-world electronic health records . |
| Outcome: | The proposed model covers five core medical areas and includes 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. |
Inductive Linguistic Reasoning with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. |
| Approach: | They propose to generate analogical exemplars with a language model and apply them in-context with target language exemplar. |
| Outcome: | The proposed method can be applied to other tasks present in Linguistics Olympiad competitions and achieves state-of-the-art results across nearly all problem types and difficulty levels in the LINGOLY dataset. |